New York City Airbnb Open Data - Exploratory Analysis and NLP

UC Davis | STA160 Group 7 | Midterm Project

• Vishnu Rangiah, 916562849, vrrangiah@ucdavis.edu
• Aditya Kallepalli, 915079375, arkallepalli@ucdavis.edu
• Rishi Bhuva, 915218518, rdbhuva@ucdavis.edu
• Ivan Yang, 915463046, igyang@ucdavis.edu

Introduction

In this report we will be analyzing Airbnb data from the Airbnb open new york city 2019 data set.

Airbnb is a platform that operates an online marketplace for lodging, primarily homestays for vacation rentals, and tourism activities. This platform is accessible through website or mobile app.

The data we are currently analyzing comes from : https://www.kaggle.com/dgomonov/new-york-city-airbnb-open-data

This dataset describes listing activity and metrics in New York City, New York for 2019. Through this dataset, we can get information regarding hosts, geographical availability and necessary metrics needed to make predictions and draw conclusions.

We will be performing our analysis using Python. In Section II, we will be cleaning, and preforming exploratory analysis on the data. Then we will explore the relationship between the names of each listing and the popularity of the listing using Natural Language Processing Methods in Section III.

NEXT STEPS: EDA, NLP, CONCLUSION

Importing Data Set

Here we find that our dataset contains 48,895 rows which represent indivual listings in New York. We are also provided with 16 different columns which contain necessary factors needed for exploratory data analysis.

We can see these different columns below:

No. of Variables + Entries, Variable Names, and Data Types:

Data Types of Each Column

A brief analysis as to what data type each column is can be seen below:

Data Cleaning

• Find outliers

• Get rid of listing that are too expensive

  • Why is the price so high: super bowl?

• Get rid of listing that do not have avalibility for 2019

• Possibly find potential negative and NA values

• The Airbnb calendar for a listing does not differentiate between a booked night vs an unavailable night, therefore these bookings have been counted as "unavailable". This serves to understate the Availability metric because popular listings will be "booked" rather than being "blacked out" by a host.

• avaliblity_365 : number of days the property is avalibly for rent

  • Data shows alot of properties of 0 days avalible
    • Why was it 0:
      • host didnt want to rent or host took off listing but it was still on sight
      • airbnb suspended listing

Our first line of action when analyzing this data is to properly clean the dataset. Having clean data will provide us with the highest quality of information needed and therefore will provide us with the most accurate predictions and correlations. Our process for cleaning the dataset can be seen below:

We will first check the number of rows with missing values. We will then remove such rows depending on the missing variables.

As we can see, there are 4 variables with missing values. However, we've determined that all of these missing variables do not greatly affect our NLP modelling. Therefore, we've decided to keep these listings.

On the other hand, while combing through our main dataframe, we discovered that there are a number of listings with 0 available days to be rented as well as 0 total reviews.

The dataframe above consists of listings with zero availability. We chose to extract these listings from the dataframe we will analyze, because zero availablity means that the listings is not available at all. Therefore, this data would be irrelevant as we are want to perform analysis on listings that have availablity.

The dataframe above consists of listings with zero number of reviews. We chose to extract these values from the data we want to analyze, because number of reviews is a factor that can tell us a lot about the popularity of the listing and is an important component to our Natural Language Processing Method which we will perform. Therefore, listings with zero number of reviews would be irrelevant data and not needed for our type of analysis.

Due to the drawbacks discussed regarding 0 days of availabilities and 0 number of reviews, it would be best to simply remove these entries in the dataset we want to work with. Below, we properly remove these listings.

Above, we can see the dimensions of our reduced dataframe, excluding values of avaiablity and number of reviews being 0.

Now, we will examine the price range of all AirBnB listings by separating these listings into quartiles to get a general insight for this column alone. Along with the 25, 50, 75 % quartile calculations, we can also get information such as the mean, standard deviation and the minimum and maximum value of our price category.

As observed in our quartile table above, there are interestingly listings that are $0. Our next course of action would be to pull up these listings and do further research.

After multiple online searches of these listings, we've come up with mixed results of these listings' robustness. Listings such as Kimberly's seem to have been removed while all three of Adeyemi's listings are still up on AirBnB and running in 2021. Additionally, there isn't any way that we can revisit AirBnB's 2019 websites to verify these listings.

One possible reason for these $0 prices may have been a deliberate act by the hosts to temporarily remove the listing from the AirBnB market when these listings were webscraped. The additional fact that among these 0 USD listings, 3 and 2 of them belong to the same host may further indicate that it is a host-activated anomaly.

Either way, we will remove these listings just to minimize any possibility of accruing errors based on unknown anomalies.

Below, is the process of removing entries with the price = 0.

Above, we can see our new summary statistics with $0 of price removed.

We can see that the data is quite skewed with multiple extremely high prices heavily affecting the data. Further research also shows that some of these data points are faulty listings, such as a 2012 Superbowl private room listing.

We will use the 1.5 Interquartile Range Test to determine outliers and store these listings in a different dataframe for further data analysis.

Here, through the IQR test, we can calculate for outliers within the price entry of data. We can see that anything above the price of 332.5 dollars would be considered too expensive, and therefore would be an outlier. Since there technically cannot be any negative values for prices, we will fix the lower limit at 10 dollars which is the minimum value as seen in our quartile table.

Tidied Dataset

Below, we have cut all outliers and properly cleaned the dataset and can move on to exploratory data analysis. We can also see the dimensions of our cleaned up dataset.

Here, we can see a proper correlation matrix between the variables within our dataset. According to this plot, we can see that the more lighter the color, the higher the correlation between each variable. Vice versa, we can see that the darker the number, the lower the correlation. From this plot, we can immediately see a high correlation of 0.6 between id and host_id. Along with this, it can be seen that number of reviews and reviews per month also have a higher correlation than others at 0.49. Along with positive correlations, we can also see negative correlations with id and number of reviews at -0.45 and with id and reviews per month at -0.45. We can also see a negative correlation between price and longitude at -0.3, although a positive correlation of 0.051 between price and latitude.

As id is a specific tag per listing, host_id is also a specific tag per host. Therefore, this positive correlation does seem to make sense as some hosts would have multiple listings under their name. Number of reviews and reviews per month also tend to have a positive correlation as each number of review contributes to a review per month and vice versa, therefore these two variables seem to have a direct relationship.

The lowest correlation we can see in this plot can be seen between both variables of reviews and id. The large negative correlation between them does seem accurate, as each id is specific and randomly generated and does not relate at all to reviews at all. Another negative correlation that was interesting comes from price and longitude. As longitude tends to be west and east, it can be seen that pricein New York does not tend to have any sort of relationship with those specific directions. Although, price and latitude show a somewhat positive correlation, meaning that price is more correlated with listings that range from the north and south rather than the west and east.

Visualizations

We will plot some graphs with our tidied dataset to gain further insight, as well as to visualize any more abnormal data points for further cleaning. We will not be subsetting these outliers from the main tidied dataset as these variables will not be our main focus for our NLP analysis. However, we will generate graphs of these outliers for further analysis.

We can see that most listings are either private rooms or the entirety of the property. This indicates that most AirBnB customers may be looking for private accomodation rather than shared spaces.

Most listings are either in the boroughs of Brooklyn or Manhattan. This is logical given that these 2 boroughs contain the most attractions and activites in New York City. Staten Island has the lowest number of listings, which may be due to its geographical location being relatively inaccessible compared to the other boroughs.

The top 4 neighborhoods of AirBnB listings are all in Brooklyn, with Bedford-Stuyvesant being the top neighborhood by quite a margin. The 5th to 8th most popular neighborhoods are in Manhattan. This may be due to affordability of Brooklyn compared to Manhattan.

Density Map of All Listings According to Price

https://geopandas.org/gallery/create_geopandas_from_pandas.html

From the density map above, we can see that most of the listings are concentrated around the boroughs of Brooklyn and Manhattan. We can also observe that Manhattan has a higher density of expensive listings, and apart from standout anomalies, prices tend to be lower as listings become further from Manhattan.

Graphs in relation to price

The plot above shows the distribution of price by room type. There is much variation in price within each room type. Through this violin plot, we can see that price greatly fluctuates between the type of room. It can be seen that entire home/apt listings tend to be more expensive than both private room and shared room, although private room is more expensive than a shared room. The logic behind the prices seem to make sense.

From the histogram above, we can see a left-skewed unimodal distribution. A majority of the listings hover around a price between 35 to 130 USD. However, there are significant spikes at 150, 200, 250, and 300 USD bins, as well as smaller spikes at 175, 225, 275, and 325 USD bins. We believe that this may be caused by human psychology where hosts may round their prices to the nearest 25 or 50 dollar for a more "aesthetically pleasing" price number.

There is an obvious trend that shared rooms are the cheapest option in all boroughs, followed by private room and entire property. Additionally, Staten Island and the Bronx have a lot less listings above 150USD compared to Brooklyn and Manhattan. Additionally, as seen in our density map, Manhattan has the highest density of expensive listings compared to other boroughs.

Most of the outlying expensive listings are either for entire properties or private rooms. Interestingly, the most expensive listing is a 10000USD private room in Queens. This is listing is now defunct, but we can infer that it might be a long term rental situation as the listing also indicates a minimum night requirement of 100 nights.

Graphs in relation to minimum nights

Most listings in 2019 NYC have a minimum night requirement between 1 to 7 nights, with an average minimum night requirement of 6.64 and 2 nights being the hightest. There is an apparent spike of listings with a minimum night requirement of 30 nights.

There seems to be multiple outliers, with one as extreme as 1250 minimum nights. This wide range of minimum night requirements allude to the existence of not just long term vacation rental properties but also potentially long term residence.

We will determine their outliers again using the 1.5 IQR test.

From the graph, it seems like Staten Island and the Bronx are not popular boroughs for long term rentals longer than 30 days. Further investigation of the few orange anomalies in the Bronx reveal that they are all from the same host, Sasha (Host ID: 2988712), who seems to have multiple 90-day listings around Claremont Village and Mount Hope in the Bronx.

Most longer term rentals in Brooklyn, Manhattan, and Queens are of entire properties. Interesting anomalies include the 90 day minimum night requirement listings for a shared room in Manhattan. Further investigation shows that these listings (Host IDs: 21628183 and 23184420) are both from LaGuardia Houses Public Housing Development looking for long term housemates. We believe that these listings are a result of hosts trying to bypass New York's legal barrier of subletting public housing for additional income, adding onto the high accessibility of AirBnB for anyone to put out a listing.

As expected, most extreme values of minimum night requirement are from listings in Manhattan and Brooklyn, with Queens and the Bronx each only having 1 listing and Staten Island not having any. Both listings in Queens and the Bronx seem to be long term rental leasings, just like most of the other leasings in this plot.

Graph by Reviews Per Month

The number of reviews are heavily skewed to the left with listings getting an average of 1.83 reviews per month. There are obvious outliers, with a listing getting as many as 58.5 average reviews per month.

We can see that most boroughs have a similar pattern of reviews per month, with 2 anomalies in Manhattan. Further analysis reveals that both listings (ID: 32678719 and 32678720) are by the same host Row NYC (Host ID: 244361589) which is a hotel in New York City. They may have more reviews as they may be more attractive to customers as an established hotel chain. Additionally, they may have multiple rooms available under the same listing, which would explain why they have listings getting 58.5 reviews in a span of 30 or 31 days.

Graphs by Availabile Days out of 365 Days

We can see a bimodal distribution with spkies at both ends. This tells us that on one extreme, most NYC hosts were not leasing out their place during 2019, or maybe for a duration not more than a week. On the other extreme, there were hosts that had listings available for almost the entire year of 2019.

All boroughs had availabilities rangine from 0 to 365, with not much of a differentiation between room types. It is interesting to note that a majority of Staten Island listings were available for more days of the year compared to other boroughs.

NLP

• Use nlp libraries to clean names column

• Find relationship between popularity (reviews per month) and price vs the most frequnet words that appear in names

• Find relatinship between names which contain unique characters such as emojis and East Asain symbols and their locations

  • For example listing names with East Asian Characters might reveal where Chinatown when looking at their Map density Plot

We will be exploring the name feature of the Airbnb dataset in depth using NLP techniques. With the use of the popular NLP library spacy we are able to clean the listing names and extract relevant information from them such as the most frequnctly used words in listings as well as non english symbols used in the listings.

In our analysis of the names of the listings we provide visualizations to show the relationship of the name listings across price subgroups and popularity subgroups which we equated to be the number of reviews per month.

Additionally we explore the unique listing names which contain non english symbols and plot their locations to see if there is any correlation between the language used in the listing and the listing location. This correlation may be present in listing names which contain chinese characters and are potentially located in china town.

An initial analysis of our raw dataframe's listing names shows that the words with the most frequencies are neighborhood and borough names, type of room, and general location indicators. It would make sense that Brooklyn and Manhattan are some of the most frequent words because they are the 2 most popular boroughs as seen in our EDA. We can also infer that hosts use listing names to capitalize on different features of the property, from location indicators such as "heart" and "near", number of bedrooms, and different landmarks and neighborhood names. Additionally, it is interesting to note the many different variations of phrases containing "cozy", such as "cozy room", "cozy bedroom", etc.

Why are we using the cleaned dataset for names

Reason: For the most part it takes out irrelevant listing which potenitally were not listing active in 2019. In terms of the focus of this report we ideally aim to see active listing in 2019 and compare the relationship between subgroups of price and popularity via number of review per month.

Reason: we only use the cleaned dataset because we want to only look at names that resemble the listing for airbnb in newyork during 2019. The cleaned data removes all the data that is invalid.

Cleaning name column

Clean the names for Filtering Dataset